Vast amounts of information are increasingly available, in part because of the growing development of rich sensors and information gathering devices. A growing proportion of this information can be characterized as flows or streams. In order for this information to be useful, it needs to be interpreted. Many methods currently exist to extract useful information from a single flow of a single data type, but these methods do not successfully combine flows of different data types for situational evaluation in near real time and on a large scale.
Generally, a multiprocessor computing environment accomplishes tasks by utilizing connected multiple processors as computing elements. The connection can be by many means, ranging from discrete processing devices that are networked together to those that have processors more integrally connected within a single device. Often, the processors consist of diverse types, optimized for different functions. For example, some processors may be optimized for video feature extraction, while others may be optimized for general reasoning. These different processors may perform analysis and transformations on very different data types.
Multiprocessor systems divide the work of the system into tasks that each processor handles. Processors communicate with one another to determine the nature of the tasks, coordinate the execution of those tasks and synthesize the results. Situationally aware multiprocessor systems are multiprocessor systems that can adjust the processes that each processor executes, change the way that tasks are defined and assigned, and adjust the communication mechanisms based on the result of computations across processes. Such an in-process synthesized multiprocessor result characterizes a “situation” that guides future tasks.
Multiprocessor systems can include parallel processing, distributed processing and virtual machines. Processors can be single or grouped and can include processing units such as those found in computers, mobile devices, gaming consoles, video cameras, military sensors, medical sensors or other devices with computing elements or logical devices.
While situationally aware multiprocessor systems provide the advantage of dynamically adapting to new information, there are many drawbacks, disadvantages and deficiencies. Processing tasks can be generally divided into: (a) those that do the basic work, for example watching a video stream for a specific behavior; (b) those that communicate partial situated information (meaning information that is relevant to the task of a single processor but created by another processor); and (c) those that look at the big picture, coordinating the work. If not properly designed, the latter two processing task types in situated multiprocessor systems grow very quickly in computational cost. This limits the scale of the system and the speed with which combined results can be provided to a user.
In the field of information processing, there are methods for transforming information variously known as stream processing, event processing, complex event processing or event stream processing. These systems are designed to deal with continuous flows of information. Examples are sensors (or otherwise supplied streams of metrics) controlling a manufacturing operation or processing plant; monitors that watch financial markets; sensors used in a military intelligence environment; and systems that watch multiple television stations and read internet-conveyed feeds, extracting patterns of interest. These systems may use specialized processors or be hosted on conventional computers or multiprocessor systems, and may include different data types in their streams. Such stream processing systems can ingest many streams; they work by inserting processors in these streams producing possibly both discrete data and transformed streams.
Processors in such systems can interact with others producing discrete and continuous data about a complex situation. While these systems provide distinct advantage in handing streams and events, they generally cannot adapt quickly. The reason is the assignment of processors and tasks to streams is currently pre-programmed in a well defined manner that is largely constrained by the types of information they are programmed to handle. The difficulty of adapting to new situations is amplified in stream processors compared to general multiprocessor systems. In other words, these systems do not reprogram themselves based on local, unexpected situations in a significant, cost-effective and scalable way.
Teachings in the field of situated reasoning can be helpful in analyzing flows of information. The teachings in this field go by many names, including system engineering; reasoning in context, modal logic, non-monotonic reasoning, and situation theory. A fundamental idea is that considering the system or situation as a whole adds new knowledge to many of the constituent processes. The fundamental mechanism is that information flowing from each process can be combined into one or more situations, some of them relevant situations that can influence a given process. One example taken from military intelligence involves the following: one sensor reports rustling in trees, another a motor sound, a third source of information is a discrete report, and a fourth is a memo reporting a stolen tank. None of these by themselves would flag a threatening situation, especially if tanks traveling under a jungle canopy is unanticipated; together they can be fused to produce a likely conclusion.
In a dynamic stream processing system, the actual depth of a process can be adapted in real time based on the situated reasoning extracted from other processors. For example, the processor or processors associated with an airborne radar can have processing tasks adjusted to analyze in more depth and to add certain vehicle recognition algorithms.
There are several methods for reasoning about and designing logical systems for such situated reasoning. The most comprehensive is situation theory, a formal system of logic designed to address this generic problem. Situation theory provides formal tools that can be applied to situated reasoning systems. A problem is that there has been no practical implementation of situation theory in processing systems because proposed architectures carry too much logical overhead, among other things.
In the field of feature processing, a “feature” is a structured piece of information that has been extracted from a stream or data item and that has utility. A simple example of a feature is used in object recognition from video. In this example, a processor associated with a video camera looks at frames of its video comparing it to certain references with a goal of identifying a tank. In doing so, it will find characteristic visual patterns (as features) that indicate the possibility that the image includes a tank. That possibility can later be reported as a semantically expressed fact, possibly with an associated probability.
The general notion of feature abstraction is widely used in many computerized applications, under different names. (Common allied concepts are Kalman filtering, Bayesian inferencing and Markov chaining.) For example, in natural language recognition, a processor will seek certain grammatical and semantic structures than can be said to be features in this way.
One use of feature structures is in the creation of b-frames for video compression. When a video compressor finds a feature or feature structure that spans multiple frames of the video, and it seems to change from video frame to video frame in an ordered manner, then it can be identified with a data structure known as a b-frame, which is a single representation for all those frames in the compressed file. On decoding, the b-frame is uncompressed once, then placed with suitable distortion in the video frames. An example may be where a camera is stationary with mountains in the background and a car passes in the foreground. One b-frame may be formed of the mountain background which is compressed and saved once for the duration of the video, using component features characteristic of a mountain and representable by the compressor as mathematical functions. A second b-frame would encode the car, which is similarly encoded and stored once, but placed and distorted on the video frames differently for each video frame, also using mathematical functions to encode the placements and distortions. Thus, the b-frames in video compression are single representations that can be used in multiple video frames, with or without alteration by mathematical formulas across frames.
In video compression, features that are more important to comprehension have been identified by a motion picture experts group (MPEG). These are both the features that convey semantic meaning when interpreted by a viewer and that provide the basic mathematical structures for representation in the compressed stream.
Representations such as b-frames are very useful when applied to processor-specific stream processing because they can be scalably implemented, be extracted quickly and the abstraction can be flexibly defined. A difficulty is that existing approaches are strongly domain-specific. In the current art, there is no specified layer of abstraction that can unify something like b-frames across processes of the same type so that processes can share constructive semantics. Moreover, in the current art, there is a lack of a similarly defined abstraction layer that can allow b-frame constructed semantic exchange across diverse stream types (audio, video, text strings, MRI output, SAR feeds and so on) to support the associated interpretive processes.
In view of the foregoing, there also exists a need in the art for a scalable system capable of collaborative analysis from different processes on different data sources. To do so, the system should create lightweight temporary data structures in order to allow communication of interim results among processes, allow shared construction of relevant situations and allow construction of semantically appropriate reports to users. The data structures should support near real time analysis, allow growth to large scale systems and accept diverse streaming inputs.